{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "M98bqxdMsTXK"
   },
   "source": [
    "# Collector\n",
    "\n",
    "The Collector serves as the orchestration layer between the policy (agent) and the environment in Tianshou's architecture. It manages the interaction loop, persists collected experiences to a replay buffer, and computes episode-level statistics. This module is fundamental to both training data collection and policy evaluation workflows."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OX5cayLv4Ziu"
   },
   "source": [
    "## Core Applications\n",
    "\n",
    "The Collector supports two primary use cases in reinforcement learning experiments:\n",
    "1. **Training**: Collecting interaction data for policy optimization\n",
    "2. **Evaluation**: Assessing policy performance without learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Z6XKbj28u8Ze"
   },
   "source": [
    "### Policy Evaluation\n",
    "\n",
    "Periodic policy evaluation is essential in deep reinforcement learning (DRL) experiments to monitor training progress and assess generalization. The Collector provides a standardized interface for this purpose.\n",
    "\n",
    "**Setup**: A Collector requires two components:\n",
    "1. An environment (or vectorized environment for parallelization)\n",
    "2. A policy instance to evaluate"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "editable": true,
    "id": "w8t9ubO7u69J",
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "hide-cell",
     "remove-output"
    ],
    "ExecuteTime": {
     "end_time": "2025-10-26T21:59:25.914405Z",
     "start_time": "2025-10-26T21:59:22.196044Z"
    }
   },
   "source": [
    "import gymnasium as gym\n",
    "import torch\n",
    "\n",
    "from tianshou.algorithm.modelfree.reinforce import ProbabilisticActorPolicy"
   ],
   "outputs": [],
   "execution_count": 1
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-10-26T21:59:30.621207Z",
     "start_time": "2025-10-26T21:59:25.922401Z"
    }
   },
   "source": [
    "from tianshou.data import Collector, CollectStats, VectorReplayBuffer\n",
    "from tianshou.env import DummyVectorEnv\n",
    "from tianshou.utils.net.common import Net\n",
    "from tianshou.utils.net.discrete import DiscreteActor\n",
    "\n",
    "# Initialize single environment for configuration\n",
    "env = gym.make(\"CartPole-v1\")\n",
    "\n",
    "# Create vectorized test environments (2 parallel environments)\n",
    "test_envs = DummyVectorEnv([lambda: gym.make(\"CartPole-v1\") for _ in range(2)])\n",
    "\n",
    "# Configure neural network architecture\n",
    "assert env.observation_space.shape is not None  # for mypy\n",
    "preprocess_net = Net(\n",
    "    state_shape=env.observation_space.shape,\n",
    "    hidden_sizes=[\n",
    "        16,\n",
    "    ],\n",
    ")\n",
    "\n",
    "# Initialize discrete action actor network\n",
    "assert isinstance(env.action_space, gym.spaces.Discrete)  # for mypy\n",
    "actor = DiscreteActor(preprocess_net=preprocess_net, action_shape=env.action_space.n)\n",
    "\n",
    "# Create policy with categorical action distribution\n",
    "policy = ProbabilisticActorPolicy(\n",
    "    actor=actor,\n",
    "    dist_fn=torch.distributions.Categorical,\n",
    "    action_space=env.action_space,\n",
    "    action_scaling=False,\n",
    ")\n",
    "\n",
    "# Initialize collector for evaluation\n",
    "test_collector = Collector[CollectStats](policy, test_envs)"
   ],
   "outputs": [],
   "execution_count": 2
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wmt8vuwpzQdR"
   },
   "source": [
    "### Evaluating Untrained Policy Performance\n",
    "\n",
    "We now evaluate the randomly initialized policy across 9 episodes to establish a baseline performance metric:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9SuT6MClyjyH",
    "outputId": "1e48f13b-c1fe-4fc2-ca1b-669485efdcae",
    "ExecuteTime": {
     "end_time": "2025-10-26T21:59:31.362074Z",
     "start_time": "2025-10-26T21:59:30.752198Z"
    }
   },
   "source": [
    "# Collect 9 complete episodes with environment reset\n",
    "collect_result = test_collector.collect(reset_before_collect=True, n_episode=9)\n",
    "\n",
    "collect_result.pprint_asdict()"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CollectStats\n",
      "----------------------------------------\n",
      "{   'collect_speed': 288.36823267420584,\n",
      "    'collect_time': 0.5860562324523926,\n",
      "    'lens': array([15, 22, 29, 22,  8, 16, 28, 10, 19]),\n",
      "    'lens_stat': {   'max': 29.0,\n",
      "                     'mean': 18.77777777777778,\n",
      "                     'min': 8.0,\n",
      "                     'std': 6.876332643007022},\n",
      "    'n_collected_episodes': 9,\n",
      "    'n_collected_steps': 169,\n",
      "    'pred_dist_std_array': array([[0.49482444],\n",
      "       [0.49513358],\n",
      "       [0.491721  ],\n",
      "       [0.49804375],\n",
      "       [0.48936436],\n",
      "       [0.49519676],\n",
      "       [0.49186328],\n",
      "       [0.4981152 ],\n",
      "       [0.49512368],\n",
      "       [0.49527684],\n",
      "       [0.49800068],\n",
      "       [0.4982014 ],\n",
      "       [0.49516457],\n",
      "       [0.4953748 ],\n",
      "       [0.49805665],\n",
      "       [0.49269873],\n",
      "       [0.49946678],\n",
      "       [0.49553478],\n",
      "       [0.49997097],\n",
      "       [0.49289048],\n",
      "       [0.4998387 ],\n",
      "       [0.4957225 ],\n",
      "       [0.49908724],\n",
      "       [0.4930759 ],\n",
      "       [0.49776313],\n",
      "       [0.49590224],\n",
      "       [0.49913287],\n",
      "       [0.4986142 ],\n",
      "       [0.4998805 ],\n",
      "       [0.49605882],\n",
      "       [0.49581137],\n",
      "       [0.49861884],\n",
      "       [0.4922438 ],\n",
      "       [0.4962572 ],\n",
      "       [0.49569792],\n",
      "       [0.49863097],\n",
      "       [0.4982123 ],\n",
      "       [0.49961406],\n",
      "       [0.49553847],\n",
      "       [0.4999985 ],\n",
      "       [0.49808985],\n",
      "       [0.4997094 ],\n",
      "       [0.49964666],\n",
      "       [0.4987858 ],\n",
      "       [0.49796286],\n",
      "       [0.4948797 ],\n",
      "       [0.49960598],\n",
      "       [0.4916098 ],\n",
      "       [0.4999896 ],\n",
      "       [0.49003887],\n",
      "       [0.4997966 ],\n",
      "       [0.48927104],\n",
      "       [0.4999768 ],\n",
      "       [0.4899478 ],\n",
      "       [0.49948972],\n",
      "       [0.49140957],\n",
      "       [0.4978501 ],\n",
      "       [0.49466696],\n",
      "       [0.49509352],\n",
      "       [0.49118617],\n",
      "       [0.49797186],\n",
      "       [0.49447665],\n",
      "       [0.49950802],\n",
      "       [0.49740306],\n",
      "       [0.498081  ],\n",
      "       [0.49935713],\n",
      "       [0.49534237],\n",
      "       [0.49994358],\n",
      "       [0.49823537],\n",
      "       [0.499905  ],\n",
      "       [0.4955164 ],\n",
      "       [0.49991024],\n",
      "       [0.49839276],\n",
      "       [0.4999328 ],\n",
      "       [0.49570385],\n",
      "       [0.4993451 ],\n",
      "       [0.49855497],\n",
      "       [0.49995714],\n",
      "       [0.4995841 ],\n",
      "       [0.49939576],\n",
      "       [0.49999622],\n",
      "       [0.49824795],\n",
      "       [0.49972966],\n",
      "       [0.49653304],\n",
      "       [0.49880832],\n",
      "       [0.49425703],\n",
      "       [0.49974373],\n",
      "       [0.49662435],\n",
      "       [0.49473634],\n",
      "       [0.49472553],\n",
      "       [0.49773476],\n",
      "       [0.49140546],\n",
      "       [0.49935693],\n",
      "       [0.48954245],\n",
      "       [0.4999408 ],\n",
      "       [0.491403  ],\n",
      "       [0.49988943],\n",
      "       [0.49483237],\n",
      "       [0.49920663],\n",
      "       [0.49134007],\n",
      "       [0.49793968],\n",
      "       [0.4894678 ],\n",
      "       [0.49924025],\n",
      "       [0.4912263 ],\n",
      "       [0.4945435 ],\n",
      "       [0.49469063],\n",
      "       [0.49759832],\n",
      "       [0.49754107],\n",
      "       [0.49466005],\n",
      "       [0.49943802],\n",
      "       [0.4977191 ],\n",
      "       [0.49995443],\n",
      "       [0.49479046],\n",
      "       [0.49937534],\n",
      "       [0.49785116],\n",
      "       [0.49731255],\n",
      "       [0.49934685],\n",
      "       [0.4993554 ],\n",
      "       [0.49798217],\n",
      "       [0.4999266 ],\n",
      "       [0.4993439 ],\n",
      "       [0.49931702],\n",
      "       [0.49815634],\n",
      "       [0.49991363],\n",
      "       [0.4993506 ],\n",
      "       [0.49928144],\n",
      "       [0.49821213],\n",
      "       [0.4973895 ],\n",
      "       [0.49938264],\n",
      "       [0.4992856 ],\n",
      "       [0.4999623 ],\n",
      "       [0.49991205],\n",
      "       [0.49940434],\n",
      "       [0.49991933],\n",
      "       [0.49825713],\n",
      "       [0.49990463],\n",
      "       [0.49554875],\n",
      "       [0.49924377],\n",
      "       [0.49196848],\n",
      "       [0.49991465],\n",
      "       [0.48965713],\n",
      "       [0.49991086],\n",
      "       [0.4888782 ],\n",
      "       [0.49921995],\n",
      "       [0.48808664],\n",
      "       [0.49516302],\n",
      "       [0.48725367],\n",
      "       [0.49179506],\n",
      "       [0.4879356 ],\n",
      "       [0.4952572 ],\n",
      "       [0.48861024],\n",
      "       [0.49187768],\n",
      "       [0.48927858],\n",
      "       [0.4953288 ],\n",
      "       [0.48839873],\n",
      "       [0.49193203],\n",
      "       [0.49538046],\n",
      "       [0.49808696],\n",
      "       [0.49537748],\n",
      "       [0.49810043],\n",
      "       [0.4953903 ],\n",
      "       [0.4981276 ],\n",
      "       [0.49956635],\n",
      "       [0.49998853],\n",
      "       [0.49978945],\n",
      "       [0.49897715],\n",
      "       [0.4975953 ],\n",
      "       [0.49903452],\n",
      "       [0.49765074]], dtype=float32),\n",
      "    'pred_dist_std_array_stat': {   0: {   'max': 0.4999985098838806,\n",
      "                                           'mean': 0.4965951144695282,\n",
      "                                           'min': 0.48725366592407227,\n",
      "                                           'std': 0.003376598935574293}},\n",
      "    'returns': array([15., 22., 29., 22.,  8., 16., 28., 10., 19.]),\n",
      "    'returns_stat': {   'max': 29.0,\n",
      "                        'mean': 18.77777777777778,\n",
      "                        'min': 8.0,\n",
      "                        'std': 6.876332643007022}}\n"
     ]
    }
   ],
   "execution_count": 3
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zX9AQY0M0R3C"
   },
   "source": [
    "### Baseline Comparison: Random Policy\n",
    "\n",
    "To contextualize the initialized policy's performance, we establish a random action baseline:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "UEcs8P8P0RLt",
    "outputId": "85f02f9d-b79b-48b2-99c6-36a1602f0884",
    "ExecuteTime": {
     "end_time": "2025-10-26T21:59:31.431099Z",
     "start_time": "2025-10-26T21:59:31.371074Z"
    }
   },
   "source": [
    "# Evaluate random policy by sampling actions uniformly from action space\n",
    "collect_result = test_collector.collect(reset_before_collect=True, n_episode=9, random=True)\n",
    "\n",
    "collect_result.pprint_asdict()"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CollectStats\n",
      "----------------------------------------\n",
      "{   'collect_speed': 4407.5322624798,\n",
      "    'collect_time': 0.053998470306396484,\n",
      "    'lens': array([11, 13, 15, 29, 15, 12, 15, 30, 98]),\n",
      "    'lens_stat': {   'max': 98.0,\n",
      "                     'mean': 26.444444444444443,\n",
      "                     'min': 11.0,\n",
      "                     'std': 26.16236105175657},\n",
      "    'n_collected_episodes': 9,\n",
      "    'n_collected_steps': 238,\n",
      "    'pred_dist_std_array': None,\n",
      "    'pred_dist_std_array_stat': None,\n",
      "    'returns': array([11., 13., 15., 29., 15., 12., 15., 30., 98.]),\n",
      "    'returns_stat': {   'max': 98.0,\n",
      "                        'mean': 26.444444444444443,\n",
      "                        'min': 11.0,\n",
      "                        'std': 26.16236105175657}}\n"
     ]
    }
   ],
   "execution_count": 4
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sKQRTiG10ljU"
   },
   "source": [
    "**Observation**: The randomly initialized policy performs comparably to (or worse than) uniform random actions prior to training. This is expected behavior, as the network weights lack task-specific optimization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8RKmHIoG1A1k"
   },
   "source": [
    "### Training Data Collection\n",
    "\n",
    "During the training phase, the Collector manages experience gathering and automatic storage in a replay buffer. This enables the experience replay mechanism fundamental to off-policy algorithms."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "editable": true,
    "id": "CB9XB9bF1YPC",
    "slideshow": {
     "slide_type": ""
    },
    "tags": [],
    "ExecuteTime": {
     "end_time": "2025-10-26T21:59:31.452144Z",
     "start_time": "2025-10-26T21:59:31.444096Z"
    }
   },
   "source": [
    "# Configuration for parallel training data collection\n",
    "train_env_num = 4\n",
    "buffer_size = 100\n",
    "\n",
    "# Initialize vectorized training environments\n",
    "train_envs = DummyVectorEnv([lambda: gym.make(\"CartPole-v1\") for _ in range(train_env_num)])\n",
    "\n",
    "# Create replay buffer compatible with vectorized environments\n",
    "replayBuffer = VectorReplayBuffer(buffer_size, train_env_num)\n",
    "\n",
    "# Initialize training collector with buffer integration\n",
    "train_collector = Collector[CollectStats](policy, train_envs, replayBuffer)"
   ],
   "outputs": [],
   "execution_count": 5
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rWKDazA42IUQ"
   },
   "source": [
    "### Step-Based Collection\n",
    "\n",
    "The Collector supports both step-based and episode-based collection modes. Here we demonstrate step-based collection, which is commonly used in training loops with fixed update frequencies.\n",
    "\n",
    "**Note**: When using vectorized environments, the actual number of collected steps may exceed the requested amount to maintain synchronization across parallel environments."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-fUtQOnM2Yi1",
    "outputId": "dceee987-433e-4b75-ed9e-823c20a9e1c2",
    "ExecuteTime": {
     "end_time": "2025-10-26T21:59:31.501487Z",
     "start_time": "2025-10-26T21:59:31.459140Z"
    }
   },
   "source": [
    "# Reset collector and buffer to clean state\n",
    "train_collector.reset()\n",
    "replayBuffer.reset()\n",
    "\n",
    "print(f\"Replay buffer before collecting is empty, and has length={len(replayBuffer)} \\n\")\n",
    "\n",
    "# Collect 50 environment steps\n",
    "n_step = 50\n",
    "collect_result = train_collector.collect(n_step=n_step)\n",
    "\n",
    "print(\n",
    "    f\"Replay buffer after collecting {n_step} steps has length={len(replayBuffer)}.\\n\"\n",
    "    f\"The actual count may exceed n_step when it is not a multiple of train_env_num \\n\"\n",
    "    f\"due to vectorization synchronization requirements.\\n\",\n",
    ")\n",
    "\n",
    "collect_result.pprint_asdict()"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Replay buffer before collecting is empty, and has length=0 \n",
      "\n",
      "Replay buffer after collecting 50 steps has length=52.\n",
      "The actual count may exceed n_step when it is not a multiple of train_env_num \n",
      "due to vectorization synchronization requirements.\n",
      "\n",
      "CollectStats\n",
      "----------------------------------------\n",
      "{   'collect_speed': 1529.5011711244197,\n",
      "    'collect_time': 0.03399801254272461,\n",
      "    'lens': array([], dtype=int32),\n",
      "    'lens_stat': None,\n",
      "    'n_collected_episodes': 0,\n",
      "    'n_collected_steps': 52,\n",
      "    'pred_dist_std_array': array([[0.4944575 ],\n",
      "       [0.49571753],\n",
      "       [0.49482644],\n",
      "       [0.49571693],\n",
      "       [0.49746   ],\n",
      "       [0.49228   ],\n",
      "       [0.491648  ],\n",
      "       [0.49237084],\n",
      "       [0.49931562],\n",
      "       [0.48953396],\n",
      "       [0.4949102 ],\n",
      "       [0.49022076],\n",
      "       [0.49992043],\n",
      "       [0.4921799 ],\n",
      "       [0.49171764],\n",
      "       [0.4894729 ],\n",
      "       [0.4992769 ],\n",
      "       [0.48948848],\n",
      "       [0.49497682],\n",
      "       [0.48870105],\n",
      "       [0.49763048],\n",
      "       [0.49201292],\n",
      "       [0.49787888],\n",
      "       [0.4893877 ],\n",
      "       [0.49927947],\n",
      "       [0.4955971 ],\n",
      "       [0.49943653],\n",
      "       [0.49005648],\n",
      "       [0.49780723],\n",
      "       [0.49179533],\n",
      "       [0.49995926],\n",
      "       [0.49153325],\n",
      "       [0.49928913],\n",
      "       [0.48941523],\n",
      "       [0.49986592],\n",
      "       [0.49499276],\n",
      "       [0.4999287 ],\n",
      "       [0.49152908],\n",
      "       [0.4991583 ],\n",
      "       [0.49093276],\n",
      "       [0.4998997 ],\n",
      "       [0.48936346],\n",
      "       [0.4978821 ],\n",
      "       [0.49442956],\n",
      "       [0.49992698],\n",
      "       [0.49117777],\n",
      "       [0.49921465],\n",
      "       [0.49751103],\n",
      "       [0.4992887 ],\n",
      "       [0.4893143 ],\n",
      "       [0.49991187],\n",
      "       [0.4992216 ]], dtype=float32),\n",
      "    'pred_dist_std_array_stat': {   0: {   'max': 0.499959260225296,\n",
      "                                           'mean': 0.49497732520103455,\n",
      "                                           'min': 0.4887010455131531,\n",
      "                                           'std': 0.003929081838577986}},\n",
      "    'returns': array([], dtype=float64),\n",
      "    'returns_stat': None}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "F:\\Users\\Dominik Jain\\Dev\\AI\\tianshou\\tianshou\\data\\collector.py:537: UserWarning: n_step=50 is not a multiple of (self.env_num=4), which may cause extra transitions being collected into the buffer.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "execution_count": 6
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Buffer Sampling Verification\n",
    "\n",
    "Verify that collected experiences are properly stored and can be sampled for training:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-10-26T21:59:31.517583Z",
     "start_time": "2025-10-26T21:59:31.509483Z"
    }
   },
   "source": [
    "# Sample mini-batch of 10 transitions from buffer\n",
    "replayBuffer.sample(10)"
   ],
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Batch(\n",
       "     obs: array([[-7.59119692e-04, -3.54404569e-01,  8.15278068e-02,\n",
       "                   6.34967446e-01],\n",
       "                 [ 2.03953441e-02, -5.46947002e-01,  4.59121428e-02,\n",
       "                   8.69558692e-01],\n",
       "                 [-5.53812869e-02, -3.63834441e-01,  1.84285983e-01,\n",
       "                   8.54350269e-01],\n",
       "                 [ 5.94463721e-02, -3.39802876e-02, -5.61027192e-02,\n",
       "                  -2.05838066e-02],\n",
       "                 [ 1.70439295e-02, -3.58715117e-01,  2.22064722e-02,\n",
       "                   6.39448643e-01],\n",
       "                 [ 1.51256351e-02,  2.27344140e-01,  1.95531528e-02,\n",
       "                  -2.54039675e-01],\n",
       "                 [-7.69001395e-02, -7.54580617e-01,  1.79230303e-01,\n",
       "                   1.36748278e+00],\n",
       "                 [-3.51171643e-02, -1.14145672e+00,  1.09657384e-01,\n",
       "                   1.86768615e+00],\n",
       "                 [ 2.10114848e-02,  3.47817928e-01, -1.05900057e-01,\n",
       "                  -6.93330288e-01],\n",
       "                 [-1.53460149e-02,  5.40259123e-01, -4.36654910e-02,\n",
       "                  -9.24050748e-01]], dtype=float32),\n",
       "     act: array([1, 1, 0, 1, 0, 0, 1, 1, 0, 1], dtype=int64),\n",
       "     rew: array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]),\n",
       "     terminated: array([False, False, False, False, False, False, False, False, False,\n",
       "                        False]),\n",
       "     truncated: array([False, False, False, False, False, False, False, False, False,\n",
       "                       False]),\n",
       "     done: array([False, False, False, False, False, False, False, False, False,\n",
       "                  False]),\n",
       "     obs_next: array([[-0.00784721, -0.16050874,  0.09422715,  0.36903235],\n",
       "                      [ 0.0094564 , -0.3524787 ,  0.06330331,  0.59165704],\n",
       "                      [-0.06265797, -0.5609251 ,  0.20137298,  1.1988543 ],\n",
       "                      [ 0.05876676,  0.16189948, -0.05651439, -0.33042672],\n",
       "                      [ 0.00986963, -0.5541395 ,  0.03499544,  0.93904114],\n",
       "                      [ 0.01967252,  0.03194853,  0.01447236,  0.04474595],\n",
       "                      [-0.09199175, -0.5620968 ,  0.20657995,  1.135794  ],\n",
       "                      [-0.05794629, -0.94769216,  0.1470111 ,  1.6109598 ],\n",
       "                      [ 0.02796784,  0.15431203, -0.11976666, -0.435774  ],\n",
       "                      [-0.00454083,  0.73594284, -0.06214651, -1.2301302 ]],\n",
       "                     dtype=float32),\n",
       "     info: Batch(\n",
       "               env_id: array([0, 0, 0, 1, 2, 2, 2, 2, 3, 3]),\n",
       "           ),\n",
       "     policy: Batch(),\n",
       " ),\n",
       " array([ 6,  3, 12, 31, 56, 53, 62, 60, 81, 78]))"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 7
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8NP7lOBU3-VS"
   },
   "source": [
    "## Advanced Topics\n",
    "\n",
    "### Asynchronous Collection\n",
    "\n",
    "The standard `Collector` implementation may collect more steps than requested when using vectorized environments. In the example above, requesting 50 steps resulted in 52 steps (the smallest multiple of 4 that is ≥50).\n",
    "\n",
    "For scenarios requiring precise step counts, Tianshou provides the `AsyncCollector`, which enables exact step collection at the cost of additional implementation complexity. This is particularly relevant for:\n",
    "- Strict reproducibility requirements\n",
    "- Algorithms sensitive to exact batch sizes\n",
    "- Fine-grained control over data collection\n",
    "\n",
    "Consult the [AsyncCollector documentation](https://tianshou.org/en/master/03_api/data/collector.html#tianshou.data.collector.AsyncCollector) for implementation details and usage patterns."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
